Skip to content

KSES: Preserve escaped shortcode brackets in text nodes - #14770

Open
salvatorecorsi wants to merge 3 commits into
WordPress:trunkfrom
salvatorecorsi:fix/66208-kses-html-api-compatibility
Open

salvatorecorsi wants to merge 3 commits into
WordPress:trunkfrom
salvatorecorsi:fix/66208-kses-html-api-compatibility

Conversation

@salvatorecorsi

@salvatorecorsi salvatorecorsi commented Oct 23, 2026 •

Copy link
Copy Markdown

Description

The HTML API implementation of KSES decodes escaped square brackets in ordinary text nodes. This changes WordPress shortcode behavior: do_shortcode( wp_kses_post( '[example]' ) ) invokes a registered example shortcode instead of displaying its literal spelling. The same regression affects examples inside code and pre, and restricted authors encounter it through the normal post-save filters.

Preserve the distinction between literal brackets and brackets decoded from character references. Use the existing HTML decoder to identify reference-origin brackets in the original text token, then retain those as numeric references when serializing the already-normalized text. Literal shortcode syntax, other HTML character decoding, and existing newline/NULL handling continue through the existing paths. A compact byte-string map keeps the additional storage proportional to the number of brackets.

This is a focused follow-up to the escaped-bracket TODO in #13271. It covers ordinary #text nodes; HTML attributes and atomic RCDATA (title/textarea) need separate provenance handling and are outside this patch. It does not change shortcode parsing or claim that the entire KSES migration is complete.

Trac ticket: https://core.trac.wordpress.org/ticket/66208

Reproduction

  1. Register a harmless shortcode named example whose callback returns a fixed marker.
  2. Sanitize [example] [example] with wp_kses_post() under the default WordPress bootstrap.
  3. Pass the result through do_shortcode().

Before: the callback runs twice. After: the encoded example remains literal and only the intentionally written shortcode runs. The regression tests also exercise wp_insert_post() as an author without unfiltered_html, then read and render the stored content.

Validation

Refreshed on October 23 after a normal merge of current trunk (3fb786c), with unchanged source and test hashes. Native PHP 8.3 / PHPUnit 9.6, using the workspace's serialized, isolated test-database wrapper:

  • Exact unchanged upstream source: 61 new test cases, 46 failures and 15 passing controls. All 61 cases pass after the correction (305 assertions).
  • KSES and shortcode groups: 1,041 tests, 3,242 assertions, four skips, no failures. This scope includes the new tests.
  • HTML decoder, modifiable text, input preprocessing, and post filtering classes: 572 tests, 1,383 assertions, one skip, no failures.
  • Changed-file WordPress Coding Standards and PHP 7.4+ compatibility checks pass.
  • Changed production file PHPStan analysis with the repository configuration passes.
  • Full npm run build passes; built KSES is byte-identical to the changed source.

The new static provider covers decimal, hexadecimal, named and alias references; leading zeroes; omitted semicolons; either encoded delimiter; mixed literal and escaped shortcodes; nested/unknown/non-bracket references; repeated sanitization; CRLF; PRE newlines; and NULL normalization. A separate test exercises real restricted-author post saving. The full Core suite and Docker E2E have not been run.

Validation after trunk update (2026-11-02)

Merged trunk 040bdd6b89913c4272bd4fdc61a79e8347cee21d normally and validated head bacc5403a224c8e4334fc93122dd498e287aa240.

Merged current trunk normally and validated the resulting tree.

  • All 61 shortcode-bracket cases passed 305 assertions; the full KSES class passed 464 tests/1740 assertions.
  • Development build, scoped production PHPStan, WordPress coding standards, PHP7.4+ compatibility, syntax and net-diff checking passed.
  • Hosted build and test jobs completed successfully on this head.

The existing broader shortcode-compatibility/design discussion remains open.

Use of AI Tools

AI assistance: Yes. Tool: OpenAI Codex. Used for investigation, implementation, regression tests, local validation, and independent automated code reviews. The account owner has not performed a human review of this patch. WordPress maintainer review is required.


This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs.

Core Committers: Use this line as a base for the props when committing in SVN:

Props salvatorecorsi, dmsnell.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.


Props Bot commands

You can trigger Props Bot actions by commenting on this PR:

  • @propsbot add <username> to give props to a GitHub user. Requires write access.
  • @propsbot remove <username> to leave a user out of the props. Requires write access.
  • @propsbot refresh to refresh this list. Adding the props-bot label works too.

@github-actions

Copy link
Copy Markdown

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

  • All changes will be lost when closing a tab with a Playground instance.
  • All changes will be lost when refreshing the page.
  • A fresh instance is created each time the link below is clicked.
  • Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
    it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

@dmsnell

dmsnell commented Oct 23, 2026

Copy link
Copy Markdown
Member

@salvatorecorsi this is an issue we worked through when building the refactor. there are a lot of shaky ways to try and prevent the unescaping of square brackets, but ultimately it seemed best to leave it, for a number of reasons:

  • above all, shortcodes are escaped by doubling the brackets like [[example]]. this is the canonical way to escape them.
  • there are code paths that need to process the unescaped HTML text, which might include URLs and style attribute values. if we intentionally send in character references after unescaping the text, we risk introducing bypasses to those other guards.
  • tracking the raw bytes through the transforms in wp_kses() is fraught at best and impossible at worst. it’s fraught because we cannot confirm that a [ from the raw text which appears in the output text in the same position is the same [; it could be coincidence that after transforming some prior content, that another intentional and unrelated bracket was put in there.
  • I’m worried about performance impacts of tracking this, though I haven’t measured to know if it’s a problem.

I considered something like you show for text nodes, because those don’t currently have any processing applied. as you scope-limited this to text nodes, that’s nice, but also sets up a weird expectation where some escaped brackets are preserved and others aren’t.

by the way, I think it’s possible to speed up the detection of these inside text spans. while we might generally have some issues using a PCRE pattern to find character references, in specific cases we should be able to find comprehensive matches.

$bracket_pattern = <<<PCRE
~
	\A                          # immediately following the found &
	(?:
		(?:[lr](sqb|brack)) # named character references
		|
		#(?:
			x0*5[BD]    # hex numeric
			|
			0*9[13]     # decimal numeric
		)
	)
	\K                          # discard the results/don’t allocate
~ix
PCRE;


while ( $at < $end ) {
	$next_at = strcspn( $raw_text, '&', $at, $end_of_span );
	…

	// Filter on candidate matches.
	if ( 1 !== preg_match( $bracket_pattern, $raw_text, $match, PREG_OFFSET_CAPTURE, $at ) ) {
		$at = $next_at + 1;
		continue;
	}

	// rely on the real decoder just for these, because the semicolon rules are complicated.
	$reference = WP_HTML_Decoder::read_character_reference( … );

	…
}

anyway, that’s just a thought. it could speed up your approach, or it may not. the code I shared is probably terribly buggy, but it’s meant to convey the idea only.


for attribute values this breaks down hard because we expect them to be transformed. unless we re-engineered wp_kses() transforms (such as URL and style sanitization), I don’t see how we can track the brackets from the start to the end. what have you considered?

@salvatorecorsi

Copy link
Copy Markdown
Author

@dmsnell Thanks for explaining the earlier reasoning. I've investigated this further with Codex-assisted analysis and automated checks, including a comparison against the legacy path and a small performance benchmark.

The compatibility case I think is worth preserving is existing entity-encoded text. With a registered probe shortcode:

Input Current legacy KSES HTML API before this patch With this patch
&#91;probe&#93; No callback Callback executes No callback
[probe] Callback executes Callback executes Callback executes
[[probe]] No callback No callback No callback

I agree that doubled brackets are the documented shortcode escape. My concern is that they do not retrofit existing encoded examples, which can start executing after a restricted author's save. They also depend on the shortcode being registered: [[unknown]] retains both brackets when it is unregistered. The 61 focused tests still pass (305 assertions), including the restricted-author save case.

For ordinary text, the mapping uses bracket order, not byte positions. The current newline/NUL normalization and HTML escaping preserve that order; the original token and the same HTML decoder identify which brackets came from references. This is the specific invariant the patch relies on. It does not restore raw HTML or change attribute, URL, or CSS sanitization.

I agree that this reasoning cannot simply be extended to attributes: their filtering can remove or rewrite substrings. I do not have a sound general provenance solution for those transforms, and would not reinsert raw attribute values after sanitization. The text-only behavior is consequently partial. Allowed title/textarea remain outside scope, and further probes confirmed other unresolved normalization cases even in ordinary text: [pr&#111;be] can still become an active shortcode, and &quot; inside a quoted shortcode parameter can change argument parsing. These also occur before this patch; preserving brackets alone does not solve the entire compatibility problem.

Your performance concern is justified. On PHP 8.3.28 CLI, comparing the exact base and patched KSES sources with warmed, repeated measurements, this synthetic input:

'<p>[ ' . str_repeat( '&amp; &copy; &#169; &mdash; ', 500 ) . '</p>'

takes roughly 3.43 ms before versus 6.89 ms after, with identical output. One literal bracket triggers an unnecessary second decode of all those unrelated entities. Ordinary 9 KB prose measured about 0.71 versus 0.73 ms; these are local function microbenchmarks, not request-level timings.

I would first address that avoidable work using a cheap candidate filter followed by the real decoder, as you suggested, and measure it again. Would you consider this limited ordinary-text compatibility fix worthwhile after reducing that cost, or would you prefer a broader approach first?

@dmsnell

dmsnell commented Oct 23, 2026

Copy link
Copy Markdown
Member

@salvatorecorsi there’s a lot in your response I unfortunately find hard to read through and understand. but here is perhaps an incomplete and unsatisfactory response.

whether we could preserve these in raw text nodes is something we knew we could do. I’ve explored other text node normalization as well. what I’m most worried about in doing this is presenting a situation where some shortcodes are preserved and others aren’t — one aspect of the wp_kses() rewrite that was important to me personally was predictability. the legacy code’s behavior was highly unpredictable because it directly depended on what syntax was presented in its inputs. the rewrite should be entirely regular, presenting developers an interface they can more or less intuit and rely on (without having to remember a list of exceptions to its behavior).

I appreciate that in your compatibility chart, you note that this is a divergence in behavior for how the old KSES worked. However, I don’t think it tells the full story. It’s an important question, and I want you to know I’m thinking through this with an open mind (and my rambling is characteristic of me trying to share my thought process transparently).


there may be opportunities to explore this at the end, using your approach with counting brackets, but instead counting actual shortcodes. that might even work for attribute values, except for some situations. one glaring example are CSS selectors which might appear, for example, in data- attributes.

<div data-target="[hidden]">

That’s not a Shortcode, though &#x5B; and &#x5D; are 100% benign in those cases.

There is vague suggestion that Shortcodes may only appear in text nodes and inside attribute values. That’s vague because the shortcode parser extends beyond that. What do you think about exploring this idea:

  • immediately before each text node or attribute value flushes into the output buffer
  • detect contained shortcode syntax (as you noted, this is not a totally tractable problem due to the way that the shortcode parser currently works, with registered or sought-after shortcodes being passed to the parser)
  • if those shortcodes weren’t present in the raw input for that text node or attribute value, then escape the brackets

@salvatorecorsi

Copy link
Copy Markdown
Author

That would give text and attributes the same rule: normalization must not introduce a shortcode that the raw input didn't contain. I'll prototype the comparison at those flush points, including encoded tag names and CSS-selector attributes. The registration-dependent parser is the part I still need to work through.

@salvatorecorsi

Copy link
Copy Markdown
Author

I tried this in an isolated prototype. It catches encoded names in both text and attributes, and bracket escaping leaves the decoded CSS selector unchanged. I'd keep detection independent of registered tags so plugin activation doesn't change the saved result.

Counts still lose which occurrence was literal: ENCODED: &#91;probe&#93; LITERAL: [probe] can keep the first active and escape the second. Matching arguments doesn't resolve identical occurrences; preserving the literal one needs a link to its raw source.

There's also [pro<notallowed>be]: removing that disallowed tag creates [probe] across two text flushes. Should this PR cover those cases too, or specifically prevent new syntax within each individual text/attribute value?

@dmsnell

dmsnell commented Oct 28, 2026

Copy link
Copy Markdown
Member

I'd keep detection independent of registered tags so plugin activation doesn't change the saved result.

this is an easy escape hatch, and something I plan on proposing, but it’s a backwards-compatibility break with unknown consequences. shortcode detection is kind of a wild mess.

removing that disallowed tag creates [probe] across two text flushes

you are starting to see the complicated cases this touches. another change to shortcodes would free this up, and I think the intent is that shortcodes today only exist within complete text nodes, or inside HTML attributes, but the current shortcode parser doesn’t do that.


and this is all work meant to ensure the prevention of creating shortcodes, yet we have to deal with representing non-shortcode lookalikes.

a good metric might simply be that we create nothing that could parse as a shortcode through wp_kses() that wasn’t already there. it’s not right to escape shortcodes by using the escaped brackets (because this is how CSS selectors, for example, need to be represented); just as it’s not right to escape everything that looks like a shortcode using double brackets (because things like CSS selectors are not shortcodes, so shouldn’t be turned into them).

this all brings me back to shortcodes, where the heart of the problem is. we’re supposed to escape anything that could appear like a shortcode, meaning that if we ever expect data-selector="[hidden]", we should be writing it like data-selector="[[hidden]]", except then we have to rely on shortcode processing that content before it gets sent to a browser, or to an API response, which often doesn’t happen.

@salvatorecorsi

Copy link
Copy Markdown
Author

You're right that registration-independent detection would change compatibility, so I won't build that assumption into this patch. The prototype only checked the decoded selector value; it didn't establish that every shortcode-processing path remains compatible. I'll keep the implementation unchanged for now, until there's a rule that prevents newly created shortcode syntax without rewriting literal selector values, including when removing markup joins text across tokens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants